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DECISION ON APPEAL 
Appellants appeal under 35 U.S.C. § 134 from the Examiner's 
rejection of claims 1-4, 6, 8-12, 14, 15, 18, 20, 21, 25, and 27-33. We have 
jurisdiction under 35 U.S.C. § 6(b). We affirm. 



1 The two-month time period for filing an appeal or commencing a civil 
action, as recited in 37 C.F.R. § 1.304, begins to run from the decided date 
shown on this page of the decision. The time period does not run from the 
Mail Date (paper delivery) or Notification Date (electronic delivery). 
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STATEMENT OF THE CASE 

Appellants invented an image enhancement method to automatically 

enhance images of human faces so that that they have more pleasing 

lightness, contrast, and/or color levels. The method can also automatically 

reduce or remove any red eye artifacts in the image. 2 Claim 1 is illustrative: 

1 . An image enhancement method using face detection algorithms, 
comprising: 

automatically detecting one or more human faces in an image using 
face detection algorithms; 

automatically locating the one or more human faces in the image; and 

automatically enhancing an appearance of the entire image by using a 
mapping technique to produce the image with target levels for a mean value 
or a variation value of the pixels in the one or more human faces, wherein 
the entire image is automatically enhanced such that the pixels in the one or 
more human faces have the target levels for the mean value or the variation 
value of the pixels. 



The Examiner relies on the following prior art references to show 
unpatentability: 



Fowler 


US 5,410,618 


Apr. 25, 1995 


Acker 


US 6,009,209 


Dec. 28, 1999 


Kado 


US 6,181,806 Bl 


Jan. 30, 2001 


Schildkraut 


US 6,292,574 Bl 


Sep. 18, 2001 






(filed Aug. 29, 1997) 


Center 


US 6,680,745 B2 


Jan. 20, 2004 






(effectively filed Nov. 






10, 2000) 



2 See generally Spec. 1:18-29; Abstract. 
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1. The Examiner rejected claims 1, 2, 8-10, 3 15,21, and 27-33 under 35 
U.S.C. § 103(a) as unpatentable over Schildkraut, Kado, and Fowler 
(Ans. 4-7). 

2. The Examiner rejected claims 3,4, 11, 12, and 18 under 35 U.S.C. 

§ 103(a) as unpatentable over Schildkraut, Kado, Fowler, and Center 
(Ans. 7-8). 

3. The Examiner rejected claims 6, 14, 20, and 25 under 35 U.S.C. 

§ 103(a) as unpatentable over Schildkraut, Kado, Fowler, and Acker 
(Ans. 8-9). 

Rather than repeat the arguments of Appellants or the Examiner, we 
refer to the Briefs and the Answer 4 for their respective details. In this 
decision, we have considered only those arguments actually made by 
Appellants. Arguments which Appellants could have made but did not make 
in the Briefs have not been considered and are deemed to be waived. See 37 
C.F.R. §41.37(c)(l)(vii). 



3 Although the Examiner omitted claim 9 in this rejection (and the other 
rejections on appeal), Appellants nonetheless indicate that this claim stands 
rejected (App. Br. 3; Reply Br. 3) — a status that the Examiner confirms as 
correct (Ans. 2 & 3). We therefore presume that the Examiner intended to 
include claim 9 in this rejection since (1) it depends from claim 8, and (2) 
the claim calls for a digital image — a feature that the Examiner indicates is 
present in Schildkraut. See Ans. 13 (noting that a human being would 
typically view the digital images as shown in Figure 1 of Schildkraut) 
(emphasis added). 

4 Throughout this opinion, we refer to (1) the Supplemental Appeal Brief 
filed Aug. 9, 2007; (2) the Examiner's Answer mailed Oct. 30, 2007; and (3) 
the Reply Brief filed Dec. 31, 2007. 
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The Obviousness Rejection Over Schildkraut, Kado, and Fowler 
Claims 1, 2, 8-10, 27, 28, and 30-33 

Regarding representative claim l, 5 the Examiner indicates that 
Schildkraut discloses an image enhancement method with a face detection 
algorithm with all of the claimed subject matter except for the last limitation 
of the claim that calls for automatically enhancing an appearance of an entire 
image via a mapping technique to produce the image with target levels for a 
mean value or variation value of pixels in the detected face(s) in the image. 
The Examiner, however, cites (1) Kado for teaching automatically 
enhancing an image's brightness based on measured brightness of imaged 
human faces, and (2) Fowler as disclosing a linear mean invariant 
transformation that enhances every strip of an image. Based on these 
collective teachings, the Examiner concludes that claim 1 would have been 
obvious (Ans. 4-6). 

Appellants argue that the Examiner misconstrues Fowler since Fowler 
merely enhances the pixels of a particular vertical strip. Appellants 
emphasize that this individual enhancement on a strip-by- strip basis has no 
effect on other vertical strips. As such, Appellants contend, the entire image 
is not enhanced to make one of the strips have target levels for the mean or 
variation values contained in any one of the strips (App. Br. 10-11, 13, 14; 
Reply Br. 5-6; emphasis added). 

5 Appellants argue independent claims 1,8, 15, and 21 together as a group. 
See App. Br. 9-18. Appellants, however, present separate arguments in 
connection with independent claims 15 and 21 (App. Br. 15-17). 
Accordingly, we select (1) claim 1 as representative of a first group 
comprising claims 1, 2, 8-10, 27, 28, and 30-33, and (2) claim 15 as 
representative of a second group comprising claims 15, 21, and 29. See 37 
C.F.R. §41.37(c)(l)(vii). 
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Appellants add that neither Schildkraut nor Kado teach or suggest 
enhancing the appearance of an entire image via the mapping technique 
including the target levels for mean or variation values of the human face 
pixels recited in claim 1. According to Appellants, Schildkraut merely 
teaches red eye reduction techniques (App. Br. 12), and Kado corrects 
"patches" of an imaged human face (App. Br. 12-13), but neither reference 
enhances the entire image as claimed (Id.). 

Additionally, Appellants contend that the Examiner's combination of 
references is improper since Fowler's field of endeavor (sonar images) 
differs from that of Schildkraut and Kado (facial images). According to 
Appellants, it would be unlikely for artisans skilled in the art of facial 
recognition and image enhancement to look at the art of sonar images, let 
alone combine the references in the manner proposed by the Examiner (App. 
Br. 17-18). 

The Examiner contends that Fowler's determination of a new mean 
and variance for each strip in the image corresponds to the recited target 
levels for each strip or image region. As such, the Examiner reasons, Fowler 
enhances the entire image since all strips are enhanced (Ans. 11). In this 
regard, the Examiner emphasizes that an entire image can be enhanced even 
by altering pixels in a portion of the image (Ans. 12). In any event, the 
Examiner further notes that even if all faces in a particular image have a 
common target level of mean or variation, Fowler's teaching still applies 
since all of these faces can be considered collectively as a region of the 
image — a region that can likewise be enhanced to have such target values in 
accordance with Fowler's technique (Ans. 11-12). 
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Claims 15, 21, and 29 

Regarding representative claim 15, Appellants argue that the cited 
prior art does not disclose that the target levels for a mean or variation value 
are desirable lightness and contrast levels that are determined through a 
determination of "human visual preferences" as claimed (emphasis added). 
According to Appellants, Kado's brightness correction is not based on 
human visual preferences, but rather is performed to make the face images 
more uniform for more accurate comparisons (App. Br. 15-16; Reply Br. 6; 
emphasis added). Appellants add that while Fowler's enhancement 
improves distinguishing lofargram signals from noise, and Schildkraut 
reduces redeye effects in images, these techniques are purportedly not based 
on human visual preferences (App. Br. 16-17; Reply Br. 6-7). 

The Examiner takes the position that Kado's adjustment of lightness 
values can change contrast accordingly (Ans. 13). In any event, the 
Examiner notes that Fowler's adjusting pixel values to achieve a desired 
variance value also results in a contrast change (Id.). Lastly, the Examiner 
notes that using target values that are desirable lightness and contrast levels 
determined in accordance with human visual preferences is known in the art 
as evidenced by Appellants' own Specification. According to the Examiner, 
since a human being would be viewing the images, there would have been 
ample reason to apply this technique (Id.). 

The issues before us, then, are as follows: 

ISSUES 

(1) Have Appellants shown that the Examiner erred in finding that 
Schildkraut, Kado, and Fowler collectively teach or suggest automatically 
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enhancing the appearance of an entire image via a mapping technique to 
produce the target levels for a mean or variation value of pixels in one or 
more detected human faces in the image such that the pixels in the face(s) 
have those target levels in rejecting claim 1 under § 103? 

(2) Have Appellants shown that the Examiner erred in finding that 
Schildkraut, Kado, and Fowler collectively teach or suggest target levels that 
have desirable lightness and contrast levels determined in accordance with 
human visual preferences in rejecting claim 15 under § 103? 

(3) Is the Examiner's reason to combine the teachings of these 
references supported by articulated reasoning with some rational 
underpinning to justify the Examiner's obviousness conclusion? 

FINDINGS OF FACT 
The record supports the following findings of fact (FF) by a 
preponderance of the evidence: 

Schildkraut 

1 . Schildkraut discloses a technique for detecting eye color defects in 
digital images 10 (e.g., "redeye" effects caused by light from a camera flash 
unit). Specifically, Schildkraut' s process (1) detects skin-colored regions in 
a digital image; (2) searches the skin-colored regions for groups of pixels 
with a redeye defect color characteristic; and (3) corrects the color of those 
pixels (Schildkraut, Abstract; col. 1, 11. 5-21; Fig. 1). 

2. Schildkraut's technique fits an ellipse to individual skin color sub- 
maps to determine whether the skin-colored region in the image is a human 
face for subsequent processing (Schildkraut, col. 4, 1. 13 - col. 5, 1. 4; Figs. 2 
and 6). 
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Kado 

3. Kado discloses a system for identifying a person by (1) inputting a 
face image; (2) extracting "feature points" related to a face from the image 
(e.g., eye, nose, mouth, etc.); (3) dividing the face image into "patches" 
(e.g., triangles) in accordance with predetermined rules; (4) extracting a 
"feature amount" for each patch to describe the face image as a set of such 
feature amounts to describe a person; and (5) comparing the extracted 
feature amounts with the feature amounts of each person in a database 
(Kado, Abstract; col. 1, 1. 63 - col. 2, 1. 3; Figs. 1 and 2). 

4. According to Kado, a "feature amount" can include the average 
brightness of each patch or, alternatively, the difference in brightnesses 
between adjacent patches (Kado, col. 2, 11. 3, 4, 1 1-13). 

5. In one embodiment, Kado's system corrects brightness of the face 
image to prevent misjudgement due to a difference in the position of the 
light source in photographing. To this end, "feature amounts extracting 
section 1 6" extracts the brightness and normal vectors of each patch. By 
observing the brightness distribution over the respective parts of the face 
image, the light source's position and direction can be estimated which 
forms the basis for brightness adjustments of particular patches (Kado, col. 
7, 11. 23-52; Fig. 14). 
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Fowler 

6. Fowler discloses a method for enhancing lofargram 6 data images 
utilizing a rule base. The system takes plural sequentially-occurring vertical 
frequency strips of a lofargram data image and computes an estimate for the 
mean and variance of each vertical frequency strip. Then, a linear 
transformation utilizing a rule base is applied to each pixel of the vertical 
frequency strip to improve the clarity of the lofargram data image (Fowler, 
Abstract). 

7. Figure 1 of Fowler details this process. As shown in that figure, 
lofargram image data 2 is segmented into vertical frequency strips and the 
mean and variance are computed for each strip. The rule base then converts 
the strip's old mean and variance into a new, enhanced mean and variance 
for that strip (Fowler, col. 3, 11. 4-23; Fig. 1 (Steps 3-5)). 

8. The new mean and variance are then used to determine a scaling 
and biasing factor for the strip that enhances the strip to a visual range more 
easily viewed by the human eye (Fowler, col. 3, 11. 19-23; Fig. 1 (Step 6)). 

9. Using the scaling and biasing factors from surrounding vertical 
frequency strips, the scaling and biasing for each pixel within a strip may be 
interpolated. The enhanced strips are then stitched back together providing a 
complete, enhanced lofargram image (Fowler, col. 3, 11. 23-30; Fig. 1 (Steps 
7 and 8); col. 6, 11. 48-58). 



6 According to Fowler, "'lofargram' refers to low frequency analyzing and 
recording gram data" (Fowler, col. 2, 11. 9-10). Additionally, Fowler notes 
that a lofargram data image, such as that shown in Figure 2a, is a frequency 
versus time representation of data output from a spectrum analyzer. These 
images are utilized in sonar detection devices to record sound frequencies 
detected over time (Fowler col. 3, 11. 32-37; Fig. 2a). 
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10. Fowler refers to Steps 5 through 7 of Figure 1 as "the 
enhancement operation" (Fowler, col. 3, 11. 26-28). 

1 1 . Fowler notes that image enhancement techniques are broadly 
applied to various types of images including, among other things, x-ray 
images and photographs (Fowler, col. 1, 11. 11-20). 

Appellants ' Specification 

12. According to the Specification, "[a]t least one study has shown 
that people prefer to look at images, such as photographs and digital images, 
with certain levels of lightness and contrast, i.e., there are desirable levels 
for a mean value and/or a variation value ... of the pixel values in the face 
region" (Spec. 5:13-16). 

PRINCIPLES OF LAW 

In rejecting claims under 35 U.S.C. § 103, it is incumbent upon the 
Examiner to establish a factual basis to support the legal conclusion of 
obviousness. See In re Fine, 837 F.2d 1071, 1073 (Fed. Cir. 1988). In so 
doing, the Examiner must make the factual determinations set forth in 
Graham v. John Deere Co., 383 U.S. 1, 17 (1966). 

Discussing the question of obviousness of claimed subject matter 

involving a combination of known elements, KSR Int'l v. Teleflex, Inc., 550 

U.S. 398, 127 S. Ct. 1727 (2007), explains: 

When a work is available in one field of endeavor, design 
incentives and other market forces can prompt variations of it, 
either in the same field or a different one. If a person of 
ordinary skill can implement a predictable variation, § 103 
likely bars its patentability. For the same reason, if a technique 
has been used to improve one device, and a person of ordinary 
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skill in the art would recognize that it would improve similar 
devices in the same way, using the technique is obvious unless 
its actual application is beyond his or her skill. Sakraida [v. AG 
Pro, Inc., 425 U.S. 273 (1976)] and Anderson' s-Black Rock[, 
Inc. v. Pavement Salvage Co., 396 U.S. 57 (1969)] are 
illustrative — a court must ask whether the improvement is more 
than the predictable use of prior art elements according to their 
established functions. 

KSR, 127 S. Ct. at 1740. If the claimed subject matter cannot be fairly 

characterized as involving the simple substitution of one known element for 

another or the mere application of a known technique to a piece of prior art 

ready for the improvement, a holding of obviousness can be based on a 

showing that "there was an apparent reason to combine the known elements 

in the fashion claimed." Id. at 1740-41. Such a showing requires 

"some articulated reasoning with some rational underpinning to 
support the legal conclusion of obviousness" .... [H]owever, 
the analysis need not seek out precise teachings directed to the 
specific subject matter of the challenged claim, for a court can 
take account of the inferences and creative steps that a person 
of ordinary skill in the art would employ. 

Id. at 1741 (quoting In re Kahn, 441 F.3d 977, 988 (Fed. Cir. 2006)). 

If the Examiner's burden is met, the burden then shifts to the 

Appellants to overcome the prima facie case with argument and/or evidence. 

Obviousness is then determined on the basis of the evidence as a whole and 

the relative persuasiveness of the arguments. See In re Oetiker, 977 F.2d 

1443, 1445 (Fed. Cir. 1992). 

Two separate tests define the scope of analogous prior art: (1) 
whether the art is from the same field of endeavor, regardless of 
the problem addressed, and (2) if the reference is not within the 
field of the inventor's endeavor, whether the reference still is 



11 



Appeal 2009-0147 
Application 09/854,580 

reasonably pertinent to the particular problem with which the 
inventor is involved. 

In re Bigio, 381 F.3d 1320, 1325 (Fed. Cir. 2004). 

ANALYSIS 
Claims 1, 2, 8-10, 27, 28, and 30-33 

Based on the record before us, we find no error in the Examiner's 
obviousness rejection of representative claim 1 which calls for, in pertinent 
part, automatically enhancing the appearance of an entire image via a 
mapping technique to produce the target levels for a mean or variation value 
of pixels in one or more detected human faces in the image such that the 
pixels in the face(s) have those target levels. 

First, while Fowler determines an enhanced mean and variance for 
individual vertical frequency strips of the image data essentially on a strip- 
by-strip basis (FF 7), this enhancement ultimately enhances the appearance 
of the entire image after stitching the enhanced strips together. See FF 9 
(noting that the enhanced strips are then stitched back together providing a 
complete, enhanced lofargram image (emphasis added)). As such, even if 
we assume that Fowler's method is limited to enhancing individual strips, 
nothing in the claim precludes such a piecemeal enhancement technique that 
collectively enhances the image's overall appearance. 

In this regard, the Examiner's point that the appearance of the entire 
image can be enhanced even by altering pixels in a portion of the image 
(Ans. 12) is well taken. Even limited alterations can enhance the overall 
appearance of an image — or do just the opposite. For example, a single 
facial feature — the smile — in the famous Mona Lisa painting is regarded as 
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one of the most fascinating aspects of this work of art (and, indeed, any 
work of art). Altering just this one facial feature in a digitized version of 
this painting would surely detract from the appearance of the image as a 
whole (i.e., the entire image) as the subject's enigmatic facial expression is a 
significant aspect of the masterpiece in view of its visual effect on the 
viewer. 7 Therefore, even enhancing an image in part (e.g., using a 
piecemeal enhancement technique as in Fowler) can enhance the appearance 
of the entire image in terms of visual impact on the viewer. 

In any event, despite Fowler's enhancement technique being 
performed on a strip-by- strip basis, it is not performed in isolation as 
Appellants seem to suggest. Rather, the enhancement operation includes 
Step 7 (FF 10), namely interpolating the scaling and biasing for each pixel 
within a strip using the scaling and biasing factors from surrounding 
vertical frequency strips (FF 9; emphasis added). As Fowler indicates, these 
scaling and biasing factors enhance the strip to a visual range more easily 
viewed by the human eye (FF 8). 

The clear import of this discussion is that the appearance of the image 
as a whole is enhanced by the individual enhancement of strips comprising 
that image. That this enhancement technique of a particular strip involves 
image data from surrounding strips (FF 9) only bolsters our conclusion that 
the entire image would be enhanced using this technique. We further note 

7 See Philip Cohen, Noisy Secret ofMona Lisa 's Smile, NewScientist, June 
23, 2004, at http://www.newscientist.com/article/dn6056 (last visited Feb. 
20, 2009); see also Louvre Museum Official Website, A Closer Look: A 
Closer Look at the Mona Lisa, available at 

http://www.louvre.fr/llv/dossiers/detail_oal.jsp7CONTENT%3C%3Ecnt_id 
=10 1 34 1 98673229908&CURRENT_LLV_OAL%3C%3Ecnt_id=l 0 134198 
673229908&bmLocale=en (last visited Feb. 20, 2009). 
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that Kado at least suggests enhancing an entire image based on the relative 
brightnesses among different patches of a facial image. See FF 4 and 5. 

We see no error in combining this teaching with the other cited 
references to arrive at the claimed invention as the Examiner proposes. 
First, we see no reason why Schildkraut's image enhancement system with 
face recognition (FF 1 and 2) combined with Kado's face image 
enhancement technique (FF 3-5) would not benefit from an enhancement 
technique such as that disclosed by Fowler which likewise seeks to improve 
clarity of the image (FF 6). Although Schildkraut enhances the facial 
images via a color correction technique (FF 1) and Kado's enhancement 
pertains to an identification system (FF 3), we nonetheless see no reason 
why improving the clarity of the image in accordance with Fowler's 
teachings would not benefit the Schildkraut/Kado system to, among other 
things, enhance discerning and distinguishing visual details in the image for 
processing and viewing. In short, the Examiner's combining the respective 
teachings of the cited references is tantamount to the predictable use of prior 
art elements according to their established functions — an obvious 
improvement. See KSR, 111 S. Ct. at 1740. 

While it is true that Fowler's system is used for lofargram data images 
which are used in sonar detection devices (FF 6), we nonetheless see no 
reason why the reference's fundamental teachings could not be applied to 
visual image enhancement systems such as Schildkraut and Kado. 
Ultimately, Fowler is concerned with improving the visual quality of 
acquired image data using enhancement techniques which Fowler at least 
suggests can be "broadly applied" to various types of images, including 
photographs. See FF 1 1 . We therefore find Fowler constitutes analogous art 
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since it is in the same field of endeavor (image enhancement) and, moreover, 
reasonably pertinent to the problem Appellants are trying to solve, namely 
image enhancement. See Bigio, 381 F.3d at 1325. As such, we find the 
references reasonably combinable. Moreover, the Examiner's reason to 
combine the teachings of the cited prior art is amply supported by articulated 
reasoning with some rational underpinning to justify the Examiner's 
obviousness conclusion 

For the foregoing reasons, Appellants have not persuaded us of error 
in the Examiner's rejection of representative claim 1 . Therefore, we will 
sustain the Examiner's rejection of that claim, and claims 2, 8-10, 27, 28, 
and 30-33 which fall with claim 1. 

Claims 15, 21, and 29 
We will also sustain the Examiner's rejection of representative claim 
1 5 which calls for, in pertinent part, determining target levels that have 
desirable lightness and contrast levels in accordance with human visual 
preferences. First, we agree with the Examiner (Ans. 13) that since humans 
would be viewing the digital images in Schildkraut {see FF 1), we see no 
reason why the target levels of lightness and contrast could not be 
determined via human visual preferences as claimed, particularly since it is 
well known that people tend to prefer certain levels for pixels in the face 
region (FF 12). While Kado's brightness adjustment technique is used in 
connection with an automatic identification system (FF 3) and prevents 
misjudgement due to differences in light source positioning in photography 
(FF 5), we see no reason why this fundamental brightness correction 
technique could not be applied to other image correction paradigms based on 
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human visual preferences, such as those disclosed by Schildkraut. That 
Kado's correction technique is based on the position of the light source used 
for the photograph and its concomitant effect on the image (FF 5) — a factor 
which itself can be a "human visual preference" — only bolsters this 
conclusion. 

Furthermore, the fact that Fowler's enhancement technique is used in 
connection with visual representations of sonar data (FF 6) hardly means 
that its target level determination would not be based, at least in part, on 
human visual preferences. Fowler determines an updated mean and variance 
that is used to determine a scaling and biasing factor that enhances a strip to 
a visual range more easily viewed by the human eye (FF 8; emphasis added). 
Simply put, the very point of Fowler's enhancement technique is to account 
for "human visual preferences," namely to make the image easier to see. As 
such, we find ample reason on this record to apply Fowler's technique to the 
Schildkraut/Kado combination as the Examiner proposes. 

For the foregoing reasons, Appellants have not persuaded us of error 
in the Examiner's rejection of representative claim 15. Therefore, we will 
sustain the Examiner's rejection of that claim, and claims 21 and 29 which 
fall with claim 15. 

Other Rejections 
Likewise, we will sustain the Examiner's obviousness rejections of (1) 
claims 3,4, 11, 12, and 18 over Schildkraut, Kado, Fowler, and Center (Ans. 
7-8), and (2) claims 6, 14, 20, and 25 over Schildkraut, Kado, Fowler, and 
Acker (Ans. 8-9). We find that Appellants have not particularly pointed out 
errors in the Examiner's reasoning to persuasively rebut the Examiner's 
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prima facie case of obviousness, but merely noted that the addition of Center 
and Acker fails to cure the deficiencies of the cited prior art in connection 
with the independent claims (App. Br. 19-20). Thus, we are not persuaded 
that the Examiner erred in rejecting these claims for the same reasons 
discussed above. The rejections are therefore sustained. 

CONCLUSION 

Appellants have not shown that the Examiner erred in rejecting claims 
1-4, 6, 8-12, 14, 15, 18, 20, 21, 25, and 27-33 under § 103. 

ORDER 

The Examiner's decision rejecting claims 1-4, 6, 8-12, 14, 15, 18, 20, 
21, 25, and 27-33 is affirmed. 
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No time period for taking any subsequent action in connection with 
this appeal may be extended under 37 C.F.R. § 1.136(a)(l)(iv). 



AFFIRMED 



ELD 



HEWLETT PACKARD COMPANY 
P O BOX 272400, 3404 E. HARMONY ROAD 
INTELLECTUAL PROPERTY ADMINISTRATION 
FORT COLLINS, CO 80527-2400 
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